Partial dependence profiles and accumulated local dependence¶
from profiles import train, X_train, Y_train, X_valid, Y_valid, column_names, treatment_col
from model import train_xgb_model, train_logistic, evaluate_uplift, simple_network, check_acc_diff, check_uplift_diff, local_search_xgb
from explanations import pdp_plot_uplift, ale_plot_uplift
train.head()
This is an uplift modelling dataset with indicator variable TREATMENT coded as segment here, orignally consisting of 3 types - no communication, mens and womans communicator type. Here, for simplicity the latter two are merged into one (because we already have information about gender in other column). Columns represent (after feature eng):
- recency - Months since last purchase.
- history - Actual dollar value spent in the past year.
- mens - customer purchased Mens merchandise in the past year
- womens - customer purchased Womens merchandise in the past year
- zip_code_Surburban - 0/1 indicator if suburban
- zip_code_Rural - 0/1 indicator if rural
- zip_code_Urban - 0/1 indicator if urban
- newbie - 1 if new customer in the past twelve months
- channel_Phone - purchased by phone
- channel_Web - puchased by web
- channel_Multichannel - above or more channels
It's easy to see, that above variables are surely highly corelated (history > 0 <-> newbie etc.). Some other variables were therefore removed.
Models and their scores¶
Robust XGBoost¶
r_xgb_model = local_search_xgb(X_train, Y_train, X_valid, Y_valid, treatment_col, just_get_model=True)
check_acc_diff(r_xgb_model, "Robust XGBoost", X_train, Y_train, X_valid, Y_valid)
check_uplift_diff(r_xgb_model, "Robust XGBoost", X_train, Y_train, X_valid, Y_valid, treatment_col)
This is model found by hyper-parameters local search: Approximate local derivative of each hyper-parameter by sampling close values, cross-validate model with them and then go ascending way, loop over all parameters for some iterations until no change appears what means we hit local minimum. Models found by this method are highly robust, and as we see the scores on train and valid datasets are close (actually, here because of some random fluctuations, valid have better score). However, this cross-validation did not use accuracy or loss as metric, but had to use adjusted gains, calculated as gain of current model comparing to random model, normalized by maximum possible achievable score (visible on above plots respectively for train and valid dataset). This parameter can be also interpreted as a relative monetary profit from using this algorithm instead of random one.
pdp_plot_uplift(r_xgb_model, X_train, Y_train, treatment_col)
ale_plot_uplift(r_xgb_model, X_train, Y_train, treatment_col)
Unfortunately, we were not able to make names on plots work, but these ids correspond to those in the dataset description.
The first thing to comment should be that ALE plots are really close to PDP profiles, so model does not have (major) interactions. However this does not mean, that we can use some classifier without interactions, because here we model directly the uplift, which is difference between preditions and is itself an interaction.
Also worth noting is, that plot 11 represents Treatment, and since we substitute this variable in prediction process, explanation function cannot see any effect of tweaking this value independently (either way we just ignore it).
Indicator variables:¶
- 0 recency
- 1 history
- 2 mens
- 3 womens
- 4 zip_code_Surburban
- 5 zip_code_Rural
- 6 zip_code_Urban
- 7 newbie
- 8 channel_Phone
- 9 channel_Web
- 10 channel_Multichannel
Indicator parameters in favor of no Treatment:¶
- Nebwie (7) -0.025. So marketing campaigns have better impact on your previous customers, which is quite intuitive conclusion.
Indicator parameters in favor of Treatment:¶
- Biggest influence can be achieved on customers who bought 'womens' products (3) +0.04.
- (2) Also customers who we know that bought 'mens' products can be easier to convince (especially since this dataset is focused on campaign directed to specific gender). This does not stand in conflict with previos statement due to two reasons: any bought products mean that this person is not a "newbie" (7), and this means we still have more information about this customer to approach him in personalized manner (some products do not have any "gender" specified).
- (9) +0.01 People who previously bought products using web. This might connect to the fact that these campigns are though emails.
Indicator parameters without significant impact:¶
- Two channel types: Phone (8) and Multichannel (10), describing the way customer used to bought products previously. It can raise some concerns, that two from three exclusive variables have no impact and one have positive: Web(9).
- All one-hot encoded zip code categories; Suburban (4), Rural (5), Urban (6), describing a dwelling-place, seem to have no effect at all. This seems reasonable that each of them dont have any impact.
History¶
Sum of previously bought products - History (1), as most distinctive variable makes the model prone to overfit. Its high fluctuations also reaffirm this thesis. Intuitively, function should ascend, stairs are probably due to tree-like nature of the classifier.
Recency¶
Days since last purchase (1). Plot provides some vague flat sinusoidal fluctuations with global maximum beeing first of local maxima. Intuitively, function should be concave, aiming to find some "sweet-spot" between time when customer "just went out from the shop" and "forgot about this". This plot can be interpretted in this manner, but also with some noise at bigger values, where classifier is littlebit overfitted to remember distinctive cases. Also, this plot is a valuable asset and can be used for choosing best time to run campaign for this customer.
Simple neural network¶
nn_model = simple_network(X_train, Y_train, X_valid, Y_valid)
check_acc_diff(nn_model, "Simple neural network", X_train, Y_train, X_valid, Y_valid)
check_uplift_diff(nn_model, "Simple neural network", X_train, Y_train, X_valid, Y_valid, treatment_col)
This is overfitted small neural network model (relu, 3 layers with channels of size 50, batchnorm, learned with early stopping based on validation dataset loss), completly unable to learn robust features.
pdp_plot_uplift(nn_model, X_train, Y_train, treatment_col)
ale_plot_uplift(nn_model, X_train, Y_train, treatment_col)
Neural network model also do not seem to have any major interactions, also we got to keep in mind that this model have berely positive score on valid dataset. First thing that comes up to mind after looking at these plots is that the estimated value is different, however for our uplift score this should not be a concern -> we only sort observations in order basing on the estimations, so only relative probabilities are important.
Indicator variables¶
Patterns visible at previous model are repeated except dwelling-place. This model suggest that capaigns are less persuasive for people from more rural areas.
Recency¶
Counter-intuitively, this function is convex instead of beeing concave -> major difference between models here.
History¶
This model suggests, that the more client spends, the less important is campaign for him. This can be caused be reasoning that they are "sure buyers".
Jan Ludziejewski¶